Digital images obtained by digital cameras (or electronic cameras) may be influenced by camera shake and object shake. Camera shake occurs when a camera itself moves while an image is being shot. Object shake occurs when an object moves while an image is being shot. In order to improve the quality of images shot by a camera, it is desirable to correct both camera shake and object shake.
As techniques for correcting camera shake, optical correction and electronic correction are known. Optical correction is realized by controlling the position of a lens or a sensor (CMOS sensor or CCD sensor, etc.) in response to movements of the camera while images are being shot. Also, if the exposure time for shooting images is reduced, camera shake is suppressed. However, a short exposure time may lead to underexposure, which deteriorates image quality. Accordingly, in electronic correction, a plurality of images are obtained by continuous shooting, and such images are synthesized after alignment of the images. This suppresses the influence of underexposure. For alignment between images, KLT and a Moravec operator can for example be used.
Object shake may also be suppressed by reducing exposure time for shooting images. Accordingly, correction of object shake is realized by controlling exposure time in response to the degree of movements of objects. For example, when the degree of object movement is high (in other words, when the area within which the object moves in the image is large), the exposure time is controlled to be short. In this correction, movements of the object are calculated in accordance with, for example, an image shot in preliminary shooting that is performed immediately before the actual shooting. In addition, image deterioration caused by underexposure may be suppressed by image synthesis, similarly to the case of camera shake correction, or may also be suppressed by image processing techniques such as a noise removing process.
It is possible to calculate a movement of an object by utilizing a plurality of images obtained by continuous shooting. However, images obtained by shooting may involve camera shake components. Accordingly, if images obtained by shooting are to be utilized to detect movements of an object, camera shake components have to be removed. Thus, methods in which movements of an object are detected by removing camera shake components have been proposed.
In the first method, camera shake is suppressed by optical correction. In this case, differences between a plurality of images obtained by continuous shooting are calculated, and thereby movements of the object are detected.
In the second method, an image motion vector detection device includes a circuit to detect the vector of camera shake from image signals obtained by a video camera. The vector expressing the movement of the object is detected based on such image signals and the vector of the camera shake (see Japanese Laid-open Patent Publication No. 6-217188 for example).
Other methods for detecting movements of an object are disclosed by, for example, Japanese Laid-open Patent Publication No. 2003-134385, Japanese Laid-open Patent Publication No. 2003-143484, and Japanese Laid-open Patent Publication No. 2006-254364. Also, Japanese Laid-open Patent Publication No. 9-116855 and Japanese Laid-open Patent Publication No. 2005-203880 disclose arts related to the present application.
However, the first method requires an optical camera shake correction mechanism in a camera. Thus, the use of the first method increases the price of cameras. Also, it is not easy to increase shock resistance (or vibration resistance) of an optical camera shake correction mechanism. Accordingly, it is difficult to include an optical camera shake correction mechanism in electronic devices that are required to be shock resistant, such as mobile phones or the like.
When the second method is to be used to correct object shake, it is necessary to calculate, immediately before shooting images that are to be recorded, a movement of the object according to a plurality of images obtained by preliminary shooting, and to determine shooting parameters (such as the exposure time) that correspond to the calculated object movement. If it takes a long processing time to determine such shooting parameters, a difference will occur between the calculated object movement and the movement that the object actually makes when it is shot. In such a case, images are shot with inappropriate shooting parameters, and thus the image quality may deteriorate. Accordingly, when, for example, the shooting capacity of a camera is 30 fps, it is desired to be able to calculate object movement within 1/30 second. However, in the second method, image alignment is performed in order to remove camera shake components. In such a case, image alignment includes procedures of extracting characteristic points, procedures of tracking the extracted characteristic points, or the like, and thus a great deal of calculations have to be performed, making it difficult to perform real-time processing. Accordingly, the second method is not appropriate to correction of object shake.
As has been described above, in the conventional techniques, it has been difficult to appropriately correct object shake by low-cost configurations because detection of object movements requires a great deal of processing.